When Is Transparency Useful?
The following essay appears in the new O'Reilly book Open Government and attempts to combine and clarify some of the points I made in previous essays. It
was written in June 2009.
Transparency is a slippery word; the kind of word that, like reform, sounds good and so ends up getting attached to any random political thing that someone
wants to promote. But just as ita€™s silly to talk about whether a€oereforma€B is useful (it depends on the reform), talking about transparency in general
wona€™t get us very far. Everything from holding public hearings to requiring police to videotape interrogations can be called
a€03transparencya€Ba€"therea€™s not much thata€™s useful to say about such a large category.
In general, you should be skeptical whenever someone tries to sell you on something like a€oereforma€B or a€cetransparency.a€B In general, you should be
skeptical. But in particular, reactionary political movements have long had a history of cloaking themselves in nice words. Take the Good Government (goo-goo)
movement early in the twentieth century. Funded by prominent major foundations, it claimed that it was going to clean up the corruption and political machines
that were hindering city democracy. Instead, the reforms ended up choking democracy itself, a response to the left-wing candidates who were starting to get
The goo-goo reformers moved elections to off-years. They claimed this was to keep city politics distinct from national politics, but the real effect was just to reduce
turnout. They stopped paying politicians a salary. This was supposed to reduce corruption, but it just made sure that only the wealthy could run for office. They
made the elections nonpartisan. Supposedly this was because city elections were about local issues, not national politics, but the effect was to increase the
power of name recognition and make it harder for voters to tell which candidate was on their side. And they replaced mayors with unelected city managers, so
winning elections was no longer enough to effect change.^
Of course, the modern transparency movement is very different from the Good Government movement of old. But the story illustrates that we should be wary of
kind nonprofits promising to help. I want to focus on one particular strain of transparency thinking and show how it can go awry. It starts with something thata€™s
hard to disagree with.
Modern society is made of bureaucracies and modern bureaucracies run on paper: memos, reports, forms, filings. Sharing these internal documents with the
public seems obviously good, and indeed, much good has come out of publishing these documents, whether ita€™s the National Security Archive , whose
Freedom of Information Act (FOIA) requests have revealed decades of government wrongdoing around the globe, or the indefatigable Carl Malamud and his
scanning , which has put terabytes of useful government documents, from laws to movies, online for everyone to access freely.
I suspect few people would put a€oepublishing government documents on the Weba€B high on their list of political priorities, but ita€™s a fairly cheap project
(just throw piles of stuff into scanners) and doesna€™t seem to have much downside. The biggest concerna€"privacya€"seems mostly taken care of. In the
United States, FOIA and the Privacy Act (PA) provide fairly clear guidelines for how to ensure disclosure while protecting peoplea€™s privacy.
Perhaps even more useful than putting government documents online would be providing access to corporate and nonprofit records. A lot of political action takes
place outside the formal government, and thus outside the scope of the existing FOIA laws. But such things seem totally off the radar of most transparency
activists; instead, giant corporations that receive billions of dollars from the government are kept impenetrably secret.
Many policy questions are a battle of competing interestsa€"drivers dona€™t want cars that roll over and kill them when they make a turn, but car companies
want to keep selling such cars. If youa€™re a member of Congress, choosing between them is difficult. On the one hand are your constituents, who vote for you.
But on the other hand are big corporations, which fund your reelection campaigns. You really cana€™t afford to offend either one too badly.
So, therea€™s a tendency for Congress to try a compromise. Thata€™s what happened with, for example, the Transportation Recall Enhancement,
Accountability, and Documentation (TREAD) Act. Instead of requiring safer cars, Congress simply required car companies to report how likely their cars were to
roll over. Transparency wins again!
Or, for a more famous example: after Watergate, people were upset about politicians receiving millions of dollars from large corporations. But, on the other hand,
corporations seem to like paying off politicians. So instead of banning the practice, Congress simply required that politicians keep track of everyone who gives
them money and file a report on it for public inspection.
I find such practices ridiculous. When you create a regulatory agency, you put together a group of people whose job is to solve some problem. Theya€™re given
the power to investigate whoa€™s breaking the law and the authority to punish them. Transparency, on the other hand, simply shifts the work from the
government to the average citizen, who has neither the time nor the ability to investigate these questions in any detail, let alone do anything about it. Ita€™s a
farce: a way for Congress to look like it has done something on some pressing issue without actually endangering its corporate sponsors.
Herea€™s where the technologists step in. a€ceSomething is too hard for people?a€B they hear. a€ceWe know how to fix that.aCB So they download a copy of
the database and pretty it up for public consumptiona€"generating summary statistics, putting nice pictures around it, and giving it a snazzy search feature and
some visualizations. Now inquiring citizen can find out whoa€™s funding their politicians and how dangerous their cars are just by going online.
The wonks love this. Still stinging from recent bouts of deregulation and antigovemment zealotry, many are now skeptical about government. a€ceWe cana€™t
trust the regulators,a€B they say. a€oeWe need to be able to investigate the data for ourselves. a€B Technology seems to provide the perfect solution. Just put it
all onlinea€"people can go through the data while trusting no one.
Therea€™s just one problem: if you cana€™t trust the regulators, what makes you think you can trust the data?
The problem with generating databases isna€™t that theya€™re too hard to read; ita€™s the lack of investigation and enforcement power, and websites do
nothing to help with that. Since no onea€™s in charge of verifying them, most of the things reported in transparency databases are simply lies. Sometimes
theya€™re blatant lies, like how some factories keep two sets of books on workplace injuries: one accurate one, reporting every injury, and one to show the
government, reporting just 1 0% of them.^ But they can easily be subtler: forms are misfiled or filled with typos, or the malfeasance is changed in such a way that it
no longer appears on the form. Making these databases easier to read results only in easier-to-read lies.
Congressa€™s operations are supposedly open to the public, but if you visit the House floor (or if you follow what theya€™re up to on one of these transparency
sites) you find that they appear to spend all their time naming post offices. All the real work is passed using emergency provisions and is tucked into subsections
of innocuous bills. (The bank bailouts were put in the Paul Wellstone Mental Health Act.) Matt Taibbia€™s The Great Deranpement ieWs the story.
Many of these sites tell you who your elected official is, but what impact does your elected official really have? For 40 years, people in New York thought they
were governed by their elected officialsa€"their city council, their mayor, their governor. But as Robert Caro revealed in The Power Broker , they were all wrong.
Power in New York was controlled by one man, a man who had consistently lost every time hea€™d tried to run for office, a man nobody thought of as being in
charge at all: Parks Commissioner Robert Moses.
Plenty of sites on the Internet will tell you who your representative receives money from, but disclosed contributions are just the tip of the iceberg. As Ken
Silverstein points out in his series of pieces for Harpera€™s (some of which he covers in his book Turkmeniscam ), being a member of Congress provides for
endless ways to get perks and cash while hiding where it comes from.
Fans of transparency try to skirt around this. a€ceOK,a€B they say, a€cebut surely some of the data will be accurate. And even if it isna€™t, wona€™t we learn
something from how people lie?a€B Perhaps thata€™s true, although ita€™s hard to think of any good examples. (In fact, ita€™s hard to think of any good
examples of transparency work accomplishing anything, except perhaps for more transparency.) But everything has a cost.
Hundreds of millions of dollars have been spent funding transparency projects around the globe. That money doesna€™t come from the sky. The question
isna€™t whether some transparency is better than none; ita€™s whether transparency is really the best way to spend these resources, whether they would have
a bigger impact if spent someplace else.
I tend to think they would. All this money has been spent with the goal of getting a straight answer, not of doing anything about it. Without enforcement power, the
most readable database in the world wona€™t accomplish mucha€"even if ita€™s perfectly accurate. So people go online and see that all cars are dangerous
and that all politicians are corrupt. What are they supposed to do then?
Sure, perhaps they can make small changesa€"this politician gets slightly less oil money than that one, so la€™ll vote for her (on the other hand, maybe
shea€™s just a better liar and gets her oil money tunneled through PACs or foundations or lobbyists)a€"but unlike the government, they cana€™t solve the
bigger issue: a bunch of people reading a website cana€™t force car companies to make a safe car. Youa€™ve done nothing to solve the real problem;
youa€™ve only made it seem more hopeless: all politicians are corrupt, all cars are dangerous. What can you do?
Whata€™s ironic is that the Internet does provide something you can do. It has made it vastly easier, easier than ever before, to form groups with people and
work together on common tasks. And ita€™s through people coming togethera€"not websites analyzing dataa€"that real political progress can be made.
So far wea€™ve seen baby stepsa€"people copying what they see elsewhere and trying to apply it to politics. Wikis seem to work well, so you build a political
wiki. Everyone loves social networks, so you build a political social network. But these tools worked in their original setting because they were trying to solve
particular problems, not because theya€™re magic. To make progress in politics, we need to think best about how to solve its problems, not simply copy
technologies that have worked in other fields. Data analysis can be part of it, but ita€™s part of a bigger picture. Imagine a team of people coming together to
tackle some issue they care abouta€"food safety, say. You can have technologists poring through safety records, investigative reporters making phone calls and
sneaking into buildings, lawyers subpoenaing documents and filing lawsuits, political organizers building support for the project and coordinating volunteers,
members of Congress pushing for hearings on your issues and passing laws to address the problems you uncover, and, of course, bloggers and writers to tell
your stories as they unfold.
Imagine it: an investigative strike team, taking on an issue, uncovering the truth, and pushing for reform. Theya€™d use technology, of course, but also politics
and the law. At best, a transparency law gets you one more database you can look at. But a lawsuit (or congressional investigation)? You get to subpoena all the
databases, as well as the source records behind them, then interview people under oath about what it all means. You get to ask for what you need, instead of
trying to predict what you may someday want.
This is where data analysis can be really useful. Not in providing definitive answers over the Web to random surfers, but in finding anomalies and patterns and
questions that can be seized upon and investigated by others. Not in building finished products, but by engaging in a process of discovery. But this can be done
only when members of this investigative strike team work in association with others. They would do what it takes to accomplish their goals, not be hamstrung by
arbitrary divisions between a€cetechnologya€B and a€o3Joumalisma€B and a€cepolitics.a€B
Right now, technologists insist that theya€™re building neutral platforms for anyone to find data on any issue. Journalists insist that theya€™re objective
observers of the facts. And political types assume they already know the answers and dona€™t need to investigate further questions. Theya€™re each in their
own silo, unable to see the bigger picture.
I certainly was. I care passionately about these issuesa€"l dona€™t want politicians to be corrupt; I dona€™t want cars to kill peoplea€"and as a technologist
la€™d love to be able to solve them. Thata€™s why I got swept up in the promise of transparency. It seemed like just by doing the things I knew how to do
besta€"write code, sift through databasesa€"l could change the world.
But it just doesna€™t work. Putting databases online isna€™t a silver bullet, as nice as the word transparency may sound. But it was easy to delude myself. All I
had to do was keep putting things online and someone somewhere would find a use for them. After all, thata€™s what technologists do, right? The World Wide
Web wasna€™t designed for publishing the newsa€"it was designed as a neutral platform that could support anything from scientific publications to pornography.
Politics doesna€™t work like that. Perhaps at some point putting things on the front page of the New York Times guaranteed that they would be fixed, but that
day is long past. The pipeline of leak to investigation to revelation to report to reform has broken down. Technologists cana€™t depend on journalists to use their
stuff; journalists cana€™t depend on political activists to fix the problems they uncover. Change doesna€™t come from thousands of people, all going their
separate ways. Change requires bringing people together to work on a common goal. Thata€™s hard for technologists to do by themselves.
But if they do take that as their goal, they can apply all their talent and ingenuity to the problem. They can measure their success by the number of lives that have
been improved by the changes they fought for, rather than the number of people who have visited their website. They can learn which technologies actually make
a difference and which ones are merely indulgences. And they can iterate, improve, and scale.
Transparency can be a powerful thing, but not in isolation. So, leta€™s stop passing the buck by saying our job is just to get the data out there and ita€™s other
peoplea€™s job to figure out how to use it. Leta€™s decide that our job is to fight for good in the world. Ia€™d love to see all these amazing resources go to
work on that.
Thanks to Andy Oram and Andy Eggers for their insightful comments on earlier versions of this essay.
For more, see http://sociology.ucsc.edu/whorulesamerica/power/local.html . ■
Fast Food Nation, Eric Schlosser, Houghton Mifflin, 2001. [p. 180] ■
You should follow me on twitter here .